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ABSTRACT 

In Taiwan, the National Science Council has implemented the High Scope Program (HSP) since 2006. The 
purpose of this study was to analyze the development and effectiveness of senior high school HSP courses on 
emerging technology. This study used a course on virtual reality as an example, to investigate the influence of 
emerging technology courses on senior high school students’ attitude toward technology. Research results 
showed that among students in the experimental group, the following constructs: cognition of the importance of 
technology, performance of technology-related action, and technology career planning, had been significantly 
enhanced. This study then developed the “Virtual Reality Course Perfonnance Assessment Scale”, and 
performed cross-evaluation of course teacher and non-course teachers to confirm this scale presenting great 
content validity, internal consistency validity, and scorer reliability. This scale can provide students and teachers 
with objective assessment indicators, which can be used to understand the learning effectiveness of students. 
Lastly, this study discussed the results of this research and external evaluation, as well as provided suggestions 
for future implementation and modification of the virtual reality course. 

Keywords: emerging technology, virtual reality, attitude toward technology, cognition of the importance of 
technology, interest in learning about technology, performance of technology-related action, technology career 
planning 

INTRODUCTION 

Emerging technologies are radical innovations based on developing technologies. Because knowledge in related 
technological domains is still in continual development, emerging technologies still present high uncertainty in 
technological innovation, product development, customer demand, market scale, and innovative benefits. 
However, emerging technologies offer significant market opportunities, and may in the future become a 
completely new industry. Emerging technologies are also poised to greatly impact existing industries and 
markets, and therefore have received a great deal of emphasis from governments, research institutions, 
corporations, and investors (Liu, 2004). Technologies such as gene therapy, high-temperature superconducting 
materials, micro-robots, luminescent material, nano-material, and green technology are all emerging 
technologies that will significantly influence humanity and society in the future. 

In Taiwan, in order to train up a skilled workforce for future development of emerging technology and scientific 
research, the Department of Science Education (National Science Council, Executive Yuan) referred to the 
Japanese “Super Science High School (SSH) Project” and implemented the “Experimental Program for the 
Development of Science and Technology Courses in Senior and Vocational High Schools” (High Scope 
Program, HSP) in 2006. The purpose of the plan was to assist senior and vocational high schools in developing 
science and technology courses in prospective domains and enhance the quality of science and technology 
education in senior and vocational high schools. Through this project, the government of Taiwan intended to 
implement the provisional course outlines in 2010 senior and vocational high schools, emphasize the 
establishment of partner relationships between universities and senior/vocational high schools, and allow 
senior/vocational high schools to develop innovative science and technology courses. The purpose of these 
courses was to enhance the quality of senior/vocational high schools nationwide, provide students with a real 
learning environment, and nurture students’ scientific research capacity - such as ability to take initiative to 
explore the development process of dynamic emerging technology and the influence of technology on humanity 
(National Science Council High Scope Program, 2007). 

The senior high school used in this research (known hereafter as High School A) was a comprehensive high 
school with considerable flexibility in its curriculum. The most prominent feature of the school was its provision 
of diversified courses for students to engage in adaptive learning. In 2001, High School A became the affiliated 
high school of a national university (known hereafter as University C) and received the full support of this 
university in hardware and software facilities. In 2007, the two schools cooperated to propose the “The Endless 
Sky: Research on Application of Wireless Hypermedia to Comprehensive High School Curriculum”, which 
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received funding from the High-Scope Program of the National Science Council. This allowed the development 
of emerging technology courses on topics such as wireless Internet technology, virtual reality, and digital 
astronomy. Through course design, these institutions hoped to boost students’ passion in learning about and 
exploring emerging technologies, guide students in developing correct attitudes toward technology, and link 
such courses with university curriculum to enhance students’ career exploration and reduce their adjustment 
period upon entry into university. 

To assist in solving the problems encountered during the development of these new courses and evaluate the 
effectiveness of the courses upon implementation, curriculum experts from University C used the CIPP 
assessment model to conduct external evaluation. The purposes were to assist High School A in building 
development and instruction models for courses on emerging technology and provide reference for internal or 
external development of related technological courses. Apart from developing courses, researcher also 
combined self-evaluation mechanisms and external evaluation to fulfill the above-described predetermined 
objectives. The researcher in this study was the course developers and used the virtual reality course they 
developed as an example of course development, as well as utilized teaching experiments to analyze the course 
and its effectiveness. The purpose of this study was not only to develop a 3D virtual reality course suitable for 
helping high school students to develop basic technological implementation techniques. Researcher also aimed 
to use quasi-experimental design to rule out pre-test effects and examine the influence of the course on the 
various constructs (cognition of the importance of technology, interest in learning about technology, 
performance of technology-related action, and technology career planning) of high school students’ attitude 
toward technology. In the next stage of planning, researcher intend to incorporate instruction in other academic 
subjects, to stimulate learners’ ability for creative development and facilitate greater competitive capacity in 
meeting the challenges of the future. 

Additionally, in the external evaluation results of course development for the second year, Huang, Lin, Wang, 
and Hou. (2009) pointed out that without an objective rating standard in assessment of learning effectiveness, 
the quality of an educational product or course is very difficult to determine. Instructors may say that “Intuition 
will indicate quality; whether the methodology is strong or not can be determined from a glance”. However, if 
teachers wish to be able to clearly plan the core abilities and subject matter to be learned each week or in each 
unit, overall assessment at fixed intervals can help teachers to more closely approach the learning situations of 
students. To solve the problem of objectivity in evaluating students’ work, Huang et al. (2009) suggested that 
the scoring indicators used in various domestic multimedia contests be used as a referential basis for assessing 
student work. Based on the above-described recommendation, another purpose of this study was to develop a 
performance assessment scale suitable for use in this course, to comprehend the learning effectiveness of 
individual students. 

LITERATURE REVIEW 
1.Virtual Reality 

Virtual reality (VR) is a emerging technology that may affect human life. VR is a type of computer-based 
simulation. VR systems have the potential to allow learners to discover and experience objects and phenomena 
in ways that they cannot do in real life (Erenay & Hashemipour, 2003). The term was first proposed by VPL 
Research founder Jaron Lanier, and refers to a fabricated environment that resembles a real-life environment. In 
actuality, virtual reality is a type of illusion that is simulated by a computer and through our senses produces a 
type of feeling that is difficult to differentiate from reality. Hsu and Shin (1997) also indicated that virtual reality 
is a new domain developed from and based on computer graphics, computer simulation, user interface and 
interactive technology. The realistic interactive simulation results of virtual reality have brought computer users 
into a new epoch. Virtual reality has the following three essential elements: imagination, interaction, and 
immersion (Burdea & Coiffet, 2003). 

Virtual reality is a computer-generated image simulation. Apart from providing lifelike scenes and images, the 
scene designer can bring feelings of vividness and fun to the user (student) through imagination. Students must 
also experience the sound and light effects generated by the virtual scene and imagine themselves present in this 
virtual space, as well as interact with the virtual scene, in order to achieve the desired results. 

The scope of application of VR includes artwork, entertainment, and education (Yoon, 2010). A great number of 
artists use VR techniques to create an environment for interacting with viewers. In 1999, Camille Utterback and 
Romy Achituv created Text Rain, an interactive installation in which participants use the familiar instrument of 
their bodies to lift and play with falling letters on the screen. Other large immersive VR projects include Osmose 
and Ephemere (Yoon, 2010). The interaction and immersion characteristics of VR techniques are also suitable to 
be applied for entertainment purposes (Burdea & Coiffet, 2003). Various online games and movies have 
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extensively adopted VR techniques; for example, the movie “Avatar” converted human motion into digital data, 
and combined the data with virtual characters via motion capture techniques. Regarding educational 
applications, VR systems allow learners to interact with learning materials, providing learners with real 
experiences and opportunities to practice continually. If VR techniques can support traditional teaching 
activities, the integrated education activities can provide learners with a more real, flexible, and effective 
learning environment (Lai, Huang, Liaw, & Huang, 2009). 

2. Attitude toward technology 

In the early 1980’s, Dutch scholar on science education Jan Raat and Marc de Vries et al. conducted an 
international study on attitude toward technology titled Pupils’ Attitude Toward Technology (PATT), which 
was successively extended to other studies related to attitude toward technology and concept of technology 
(Heywood, 1998). In 1993, Jeffrey developed a simpler measurement tool, the Technology Attitude scale (TAS- 
USA) based on the PATT-USA scale. The purpose was to use this scale to test and measure students’ attitude 
toward technology. Both PATT and PATT-USA consisted of six main constructs: “personal interest in 
technology”, “role of technology”, “ results of technology”, “difficulties with technology”, “courses on 
technology”, and “career in technology”. 

In Taiwan, the constructs of scales related to attitude toward technology have been developed largely from the 
six constructs of the PATT scale. An example of this is the four constructs of the scale for attitude toward 
technology developed by Chang (1998): “content of technology”, “technological techniques”, “technology 
products”, and “influence of technology”. The scale for junior high students’ attitude toward technology 
developed by Cheng (2000) was divided into six constructs: “reaction to technological objects”, “basic 
awareness of the importance of technology”, “vigilance toward technology”, “behavioral inclinations with 
regard to technology”, “ assessment of the value of technology”, and “organizational analysis capacity regarding 
technology”. The scale for the attitude of junior high school students toward technology developed by Yu, Han, 
Hsu, and Lin (2005) was divided into five constructs: “learning about technology”, “interest in technology”, 
“difficulties of technology”, “contribution of technology”, and “technology in life”. 

Combining the above-described references, Huang, Lin, Wang, and Hou. (2008) divided attitude toward 
technology into four constructs based on the areas of cognition, skill, and affective meaning. 

• Cognition of the importance of technology: Refers to individual conception of the influence of 
technology on life and society 

• Interest in learning about technology: Refers to individual reaction to, interest in, and willingness to 
learn about new information or developments related to technology; preferences and degree of 
satisfaction with regard to learning about technology, and desire to gain an in-depth understanding of 
technology. 

• Performance of technology-related actions: Refers to the level to which individuals use actions or 
behavioral inclinations to express their attention to and preferences regarding technology-related 
objects and matters. 

• Technology career planning: Refers to individuals planning for technological study and technological 
work in their future careers. 

Concerning research related to attitudes toward technology, Dutch technology education scholars Raat and de 
Vries. (1985) surveyed 2500 eighth grade students on their attitudes toward technology. The research results 
indicate that most of the students conceived of technology as machines or facilities, stimulating the participation 
of global scholars in research on attitudes toward technology. Hurley and Vosburg (1997) explored the 
following issues: student attitudes toward emerging technology, whether these attitudes are positive, and 
whether a significant correlation exists between technology attitudes and attitudes toward emerging technology. 
The research results found that the student attitudes toward technology and learning attitudes toward emerging 
technology are positive and significantly correlated.. The research by Boser, Palmer, and Daugherty (1998) 
investigated the influence of different teaching methods used in technology education courses and their effect on 
student attitudes toward technology. The research discovered that three out of the four teaching methods have an 
impact on the student attitudes toward technology; however, in the dimension of “interest in technology”, none 
of the four teaching methods are effective. Moreover, after nine weeks of technology education courses, the 
students did not exhibit positive performance in the dimension of technological concepts. Scholars in other 
countries have also engaged in related research. Van Rensburg, Ankiewicz, and Myburgh (1999) adopted the 
PATT-USA scale to investigate attitudes toward technology among South African students aged between 12 and 
16 years. The research results confinned that technology is a vital issue of curriculum design in the near future. 
Voke and Yip (1999) translated and revised the PATT-USA scale to the PATT-HK scale. The PATT-HK scale 
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was applied to investigate the attitudes of junior high school students in Hong Kong toward technology in 1999 
and 2005. The research results indicate that altering Design and Technology courses with technology 
development can positively influence student attitudes toward technology (Voke & Yip, 2005). 

Accordingly, the PATT scale has been widely applied to research on teaching and course design regarding 
technology education since the 1980s, facilitating course designers in planning technology education courses 
that support students’ interests and needs. This study uses VR to develop emerging technology courses and the 
PATT scale to understand the impact of emerging technology courses on high school student attitudes toward 
technology. 

3. Performance Assessment 

With regard to the definition of performance assessment, Wu and Lin (1997) pointed out that performance 
assessment can also be called a non-written test. It is an assessment based on the students’ performance in actual 
completion of a specific task or piece of work. These tasks or pieces of work may be actual practice, a verbal 
report, a scientific experiment, math problems, or composition. Lu (1998) indicated that the concept of 
performance assessment and portfolio assessment, also called alternative assessment, was a hot topic in the 
fields of education and educational testing and assessment in the U.S in the 1990’s. Performance assessment can 
also be used as a strategic method for assessing the perfonnance of the school and comprehending the general 
level of student achievement. 

This study compiled the viewpoints of the above-described researchers (Wu and Lin, 1997; Lu, 1998; Stiggins, 
1987) and defined performance assessment as a method of using observation and professional judgment to 
assess the learning achievements of students. The forms of performance assessment are diverse, and can include 
constructed response questions, written report, essays, speech, practice, experimentation, data collection, and 
display of works. 

Evaluation items for the VR courses 

The VR course developed by this study includes virtual roles and 2D and 3D designs in scenes and space, with 
the aim to develop students’ art and design abilities. Therefore, this study used the evaluation standards of 
complete art and design work to develop items for evaluating actual perfonnance in the VR courses. 

Regarding works of art, the evaluation items include: (1) Completing the works on time; (2) the presentation of 
the works; and (3) learning attitudes (Liu, 2007; Hung, 2009). The items evaluating the on-time completion of 
work assess whether students are responsible. The presentation of artwork can reflect the process students in 
adjusting their physical and mental conditions; thus, an objective and multidimensional evaluation including 
five items, composition, color, technique, creativity, and completeness, was applied to cultivate the students’ 
confidence. The main purpose of learning attitudes is to assess whether students work hard, focus on their work, 
and exhibit persistent learning attitudes. Indicators of learning attitudes can affect future student work attitudes 
and results. The evaluation items for design work suggested by the teachers are listed below in order of 
importance: (1) Creativity, (2) correspondence to topics, (3) the completeness of the work, (4) function, (5) 
aesthetics, and (6) presentation techniques (Yan & San, 2008). 

Creativity is the most crucial indicator in the presentation of a work. Therefore, this study uses the evaluation 
items of artwork to evaluate design work. The researcher also increased the weight of the sub-items in creativity. 
To satisfy the requirements of this course, “composition” was replaced by “correspondence to topics” among the 
sub-items in the presentation of works. 

RESEARCH METHODS 
1. Development of Virtual Reality Course 

Composition of research team 

This group was mainly composed of teachers from the computer science course of the school being researched. 
The researcher also invited a professor from department of electrical engineering of university C (with many 
years of experience in teaching, research, and industry-academia cooperation in image processing, video 
processing, video compression, graphic theory and DSP/IC design) for guidance in professional technology and 
assessment of learning effectiveness, as well as invited three professors from a graduate insitute of curriculum 
and instructional technology at university C to assist in evaluating the course. 
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Stages of Course Development 

In the first year, a course development committee was established to explore the core knowledge of virtual 
reality and teaching material with a structure suitable for high school level. These materials were then tested 
according to suitability of teaching methods. Each chapter was developed as one unit and comrpehensively 
discussed by the course development committee. Reflective logs and meeting minutes were also kept. After the 
course development committee had discussed and modified the material, the course was finalized, and the 
textual materials were prepared at the end of the year. 

In the second year, researcher began to implement the instructional activities of the virtual reality course. With 
the counsel of the university professors, researcher used quantified experimenal design, qualitative classroom 
observation, and interviews to investigate whether the teaching materials were appropriate, the teaching methods 
suitable, and whether the course could enhance the attitude of students toward technology. The resulting data 
was used to modify the teaching materials and methods. 

In the third and fourth years, researcher repeated the implementation of the modified teaching materials and 
methods, in the hope that the course would more accurately meet the needs of high school students. Lastly, 
researcher conducted promotional activities in the fourth year, working to extend implementation of the course 
to senior and vocational high schools nationwide. 

Teaching Objectives 

The objectives of this course are as follows: 

• To enhance the learning motivation and literacy of students with regard to emerging technology 
through the instruction of an innovative course. 

• To use investigation and practical application of virtual reality to increase the degree to which students 
use actions or behavioral tendencies to express their attention to or preferences regarding technology- 
related matters. 

• To link this course with university curriculum to enhance students’ career exploration and reduce their 
adjustment period upon entry into university. 

• To train students to develop the basic abilities of industry professionals and join future technological 
industries. 

Course development model 

Organized the “virtual reality course development committee”, invited the guidance of expert university 
scholars and teachers in related domains, and used a three-stage model for course development: A (analysis); I 
(design), D (development). The committee worked to ascertain core knowledge, establish the course framework, 
arrange the order of chapters and sections, integrate vertical and horizontal links, discuss learning goals, develop 
units, write teaching material content, and solicit the opinions of experts and teachers in related domains to 
modify the material. The committee designed different forms and charts for each stage, to ensure course 
development quality. This course was developed on the basis of students’ mathematics and physics courses, 
such as coordinate systems, plane coordinate conversion, vectors, matrix, and curves in mathematics courses 
and kinematics, two-dimensional collision, and three-dimensional collision in physics courses. The virtual 
reality course framework was planned according to the principles and key elements of virtual reality. 

Course Content 

The course developed by this study includes a textbook and a teacher’s manual, and can be used as a 4-credit 
point (one academic year) elective course in senior high school. The course units include: (l)What is virtual 
reality; (2)Input and feedback devices of virtual reality; (3)Design principles of virtual reality; (4)Virtual interior 
design; (5)Design of virtual scenery; (6)Design of virtual characters, and (7)Virtual reality and 3D technology. 

Software and hardware facilities for teaching 

In consideration of factors such as funding problems in high schools and the degree of acceptance of beginners, 
researcher integrated different types of software (such as Poser, Bryce, and Space Magician) in designing this 
course, with the intention of developing students’ ability to operate and use different types of software for 
“virtual reality” digital creation. 

Poser was chosen because Poser was the first tool that could produce complete 3D character animation. 
Although 3DS is used for most professional scenery design, it is time-consuming for even the most familiar 
user. Therefore, this study used the recently increasingly popular Bryce in scenery design programs. The user- 
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friendly interface and powerful features of this program make it very suitable for beginners in virtual reality 
technology. 

In the practical part of the virtual reality course, the researcher selected Space Magician as the first step in 
practice. The reason was that this software is easily learned and easy to understand, and offers a humanized 
operation interface, specific 3D presentation of space, fast drawing, instant imaging features, easy operation, and 
absence of complex commands. This program is therefore easy for beginners to learn to use. Additionally, with 
regard to teaching facilities, because this course involves learning to use related graphics software, higher-end 
computer facilities are needed, preferably with high capacity memory and separate display chips. With regard to 
the input/output devices required for the course, high schools that wish to teach this course can, funds 
permitting, purchase input devices such as lower-priced optical data glove, shutter glasses or head-mounted 
displays (HMDs) for teaching purposes. Moreover, through school strategic alliances, high schools may also be 
able to use mid-range and high-end equipment in virtual reality lab of university, allowing students to 
experience the actual process of 3D virtual reality animation filming such as “motion capture mirrors emotions” 
how to work, is shown in Figure 1. 



Figure 1: students visited to 3D virtual reality lab of nearby university to allow them to experience the actual 

process of 3D virtual reality animation filming. 


Instructional Implementation 

Virtual reality courses are usually considered professional or advanced courses, or graduate courses, in 
university. Therefore, such courses must be adjusted according to the basic abilities of students when taught in 
high schools, to avoid learning difficulties. However, at the same time, virtual reality is a topic that can spark 
more interest among students and inspire them to take initiative to explore the development of this new 
technology, and thereby develop the technological and information-related abilities required. The instructional 
process of this course is shown in Figure 2. 
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demonstration and instruction provide with a scaffold introspection and exploration 

Figure2: Instructional process of the virtual reality course used a three-stage model for instructional 
Implementation: demonstration and instruction; provide with a scaffold, introspection and exploration. 

Evaluation of teaching effectiveness 

The PDCA cyclic model was used to construct the model of evaluation for the teaching cycle in this course. 
From the establishment of teaching goals and selection of teaching content, to implementation of teaching 
activities and evaluation of teaching results, teachers constantly reviewed the implementation of the teaching 
process and studied paths for improvement. Review meetings, classroom observation, questionnaire survey, 
teaching experiments, teacher reflection journals, and creation of teaching files were used to ensure the quality 
of course development. 

2. Experimental Design 

This study recruited 89 eleventh grade students majoring in science of High School A in 2010. To better 
understand the impact of the VR course developed by this study on the high school students’ attitudes toward 
technology, the recruited students were divided into two groups according to their course selection results. The 
experimental group was composed of 31 students who selected the VR course, and the control group 
encompassed 21 students who did not select any emerging technology courses. To avoid disruptions to internal 
validity caused by the students selecting other emerging technology courses, the 37 students who selected other 
emerging technology courses were excluded from the control group. 

Regarding the experiment design, this study used quasi-experimental research methods to investigate the 
difference in attitude toward technology between sophomore students (experimental group) who had attended 
the virtual reality course, and sophomore students (control group) who had not attended the course on emerging 
technology. Researcher conducted pre-testing in the first week of the 2010 school year, and conducted post¬ 
testing in the 1 l lh week; the teaching experiment period was thus 10 weeks. The experiment design is explained 
in Table 1. 

This study considered only the course structure of the High School A, and respected the free will of students to 
take elective courses; therefore, a true experimental design could not be conducted. The conclusions of this 
study can only be applied to students majoring in science, instead of general senior high school students. 


Table 1: Experiment design. 


Group 

Pretest Experimental Treatment 

Posttest 

Experimental Group 

Oi X 

0 2 

Control Group 

o 3 

o 4 


1. The experimental group: those who took the VR course. 

2. The control group: those without any experimental treatment. 
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3. Instruments 

Scale of high school students’ attitude toward technology 

The test instrument in this experiment was the “Scale of High School Students’ Attitude toward Technology”. 
This scale was developed by the academic evaluators of this course (Huang et al., 2008). This scale was divided 
into four constructs: “cognition of the importance of technology”, “interest in learning about technology”, 
“performance of technology-related action”, and “technology career planning”. After being tested for construct 
validity by experts and modified, the scale was developed into a pre-test questionnaire. The targets of the pretest 
were 220 high school students. After the questionnaires had been returned, a valid sample of 202 students was 
obtained, making a 91.81% questionnaire recovery rate. Researcher then conducted item analysis and factor 
analysis, and removed seven items. With regard to reliability testing, the Cronbach's a of the overall 
questionnaire on attitude toward technology was .927, and the Cronbach's a values of the four constructs were 
.712, .846, 847, and .871. These results showed that this questionnaire had high reliability. 

Performance assessment scale 

To understand the effects of the students’ actual practice and learning, this study developed a valid and objective 
performance assessment scale. Students’ work was provided to the course teacher for assessment from his/her 
professional angle. Two professional teachers who had not taught the course were invited to provide assessment, 
after which students were allowed to self-evaluate their work using the assessment scale. Statistical analysis was 
used to test the validity of the assessment scale developed by this study. In developing and modifying the 
evaluation indicators in the assessment scale, this study referred to the assessment scale developed by Liu 
(2007) and the various evaluative indicators used in assessment of entries in the design category at the National 
Commercial Art Technology Competition in Taiwan, as well as solicited the opinions of experts. 

RESEARCH RESULTS 

1. Descriptive Statistics Analysis 

Table 2 shows the mean, standard deviation, and adjusted mean corresponding to each variable as derived from 
data generated by the pre-testing and post-testing of students in the experimental group and the control group. 
Overall, students who had competed studying the virtual reality course performed better with regard to the four 
constructs of attitude toward technology, as compared to their pre-course performance. Their performance also 
exceeded that of the control group. 

2. Covariate Analysis 

Due to the limitations of class structure, this study was unable to use random allocation in research design, and 
therefore used the pre-test as a covariate to implement statistical control. This study then conducted analysis of 
covariance (ANCOVA) of the experimental group and control group. Prior to conducting ANCOVA, this study 
first conducted a test of homogeneity of group regression coefficients. The results of the interaction between the 
independent variables and the covariates were as follows: 

“Cognition of the importance of technology”: F (1, 48) =1.20, p>.05,“Interest in learning about technology”: F 
(1, 48) =.47, p>.05, “Performance of technology-related action”: F (1, 48) =.51, p>.05, “Technology career 
planning”: F (1, 48) = 01, p>.05. 

These results did not reach a level of significance, indicating that the linear relationships between the covariates 
and dependent variables of each group were consistent. 


Table 2: Mean and standard deviation of variables in attitude toward technology 


Variables 

Experimental group 
(N=31) 

Control group 
(N=21) 


M 

SD 

M 

SD 

Cognition of the importance of technology 
(pre-test) 

38.35 

3.00 

35.90 

3.66 

Interest in learning about technology 
(pre-test) 

44.45 

5.09 

44.57 

5.09 

Performance of technology-related actions 
(pre-test) 

53.81 

9.84 

53.90 

13.43 

Technology career planning 
(pre-test) 

15.23 

2.85 

15.57 

4.15 

Cognition of the importance of technology 
(post-test) 

38.29 

3.33 

33.95 

4.57 

Interest in learning about technology 

46.32 

5.86 

43.14 

7.74 
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(post-test) 

Performance of technology-related action 
(post-test) 

Technology career planning 
(post-test) 

56.68 

8.99 

49.14 

13.40 

17.16 

3.16 

14.71 

3.86 

Cognition of the importance of technology 
(adjusted) 

Interest in learning about technology 
(adjusted) 

Performance of technology-related action 
(adjusted) 

Technology career planning 
(adjusted) 

38.29 


33.96 


46.33 

43.14 

56.68 

49.14 

17.16 

14.71 


3. Development of the performance assessment scale 

Apart from using the above-described standardized measurements to assess the effectiveness of the course, this 
study also designed the “virtual reality course performance assessment scale” (see Appendix), based on 
suggestions provided by Huang et al. (2008) after her external assessment of the course. The analyses which 
follow are the validity and reliability analyses of the scale. 


Table 3: Summary of one-way analysis of covariance regarding attitude toward technology 


Source of variation 

SS 

df 

MS 

F 

T 

Cognition of the importance of technology 






Pretest(covariance) 

.01 

1 

.01 

.001 


Group 

205.48 

1 

205.48 

13.44 ** 

.22 

Error 

749.33 

49 

15.29 



Interest in learning about technology 






Pretest (covariance) 

3.35 

1 

3.35 

.08 


Group 

127.05 

1 

127.05 

2.80 

.05 

Error 

2222.00 

49 

45.35 



Performance of technology-related actions 






Pretest(covariance) 

19.29 

1 

19.29 

.16 


Group 

711.71 

1 

711.71 

5.81* 

.11 

Error 

5998.06 

49 

122.41 



Technology career planning 






Pretest (covariance) 

.09 

1 

.09 

.01 


Group 

75.04 

1 

75.04 

6.14* 

.11 

Error 

598.39 

49 

12.21 




*p<.05 "p<.01 ***p<.001 


Content Validity 

In designing the assessment scale, this study referred to the assessment scale developed by Liu (2007) and the 
various evaluative indicators used for scoring purposes in the National Commercial Art Technology 
competition. To judge whether the constructs and items of the scale were appropriate, professors from the 
Graduate Institute of Curriculum Instruction and Technology of University C were invited to review the first 
version of the scale along with professors from art departments of other schools. This study modified each 
dimension and item according to the suggestions provided by the experts, indicating that the scale has a certain 
internal validity. 

Correlation Analysis 

Six dimensions for assessment of work were developed: communication of theme, creativity, color, technique, 
structure, and comprehensiveness. Prior to assessment, researcher discussed the evaluation indicators 
corresponding to these six dimensions, and Pearson’s product moment correlation coefficient was used to 
investigate the relationships among the six dimensions and total score. Table 4 shows that no significant 
correlation existed between communication of theme and color (r =.284) or structure (r =.341). However, 
moderate and positive correlations were found to exist among the other dimensions. The research theme of this 
study was design-related courses, and the assessment indicators (creativity, technique, comprehensiveness, 
communication of theme, color, and structure) were found to be moderately and positively correlated, which 
indicates that except for color and structure, each dimension has a high correlation coefficient, meaning that 
these dimensions are highly correlated. 
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Moreover, as can been seen in Table 4, the six dimensions and the total score of the scale are significantly 
correlated, indicating a high internal consistency. 

Scorer Reliability 

Scorer reliability refers to whether different scorers demonstrated consistent assessments on the same indicator 
of performance. To assess whether the scale has scorer reliability, this study used the assessment of the course 
teacher (T A ) and the scale-based assessments of non-course teachers (T B , T c ); of these, T B is a computer teacher 
and T c is an art teacher. Student work consisted of two parts: part one was a personalized suite, and part two 
was residential design. Course teacher used his professional viewpoint for assessment, and non-course teachers 
assessed the work according to the assessment scale. To understand whether differences existed between the 
assessment scores given, this study tested for differences among the scores. 


Table 4: Correlation analysis of the six assessment dimensions 


Measure 

Communication 

Creativity 

Color 

Technique 

Structure 

Comprehensiveness 

Communication 

- 






Creativity 

.393* 

- 





Color 

.284 

.618** 

- 




Technique 

.546 

.652** 

.445* 

- 



Structure 

.341 

.667** 

.553** 

.710** 

- 


Comprehensiveness 

.505** 

.751** 

.550** 

.841** 

.764*’ 

- 

Total 

.657** 

.881** 

.713** 

.839** 

.809** 

.899*’ 


*p<.05 "p<.01 ***p<.001 


The course teacher used 100 as the full score when performing assessment. When using the assessment scale for 
evaluation, the non-course teachers first subtracted the subjective dimensions of “work finished on time/attitude 
toward learning”; therefore, the full score used was 80. The score of the non-course teachers times 100/80, then 
all of them use 100 as full marks. 

Based on the calculated scores, the researcher of this paper used Kendall’s coefficient of concordance to analyze 
the level of consistency of the three scorers on the two works. The analysis shows consistent results regarding 
the design work of “personalized suites” (Kendall’s coefficient of concordance = .680, chi-square value = 
53.056, and p < .05) and “living house” (Kendall’s coefficient of concordance = .611, chi-square value = 
47.652, and p < .05); indicating that a significant correlation exist among the three scorers. Because the 
analytical result shows consistency among the three scorers, this study infers that this scale possesses scorer 
reliability. 

DISCUSSION AND CONCLUSIONS 

The above-described analysis results showed that after 10 weeks of attending the virtual reality course, students 
in the experimental group performed better in two constructs of attitude toward technology (performance of 
technology-related action, and technology career planning) as compared to their pre-course performance. Their 
performance also exceeded that of the control group. This showed the preliminary results of the course in 
enhancing the comprehension of students with regard to knowledge of emerging technology and the 
development of technological ability. The research results of this study prove that emerging technology courses 
do not positively influence the cognition of the importance of technology; this conclusion is similar to the 
research result of Boser, Palmer, and Daugherty (1998), which found no positive correlation between 
technology courses and student concepts of technology. The researcher infers that the cause might be an 
overemphasis on the teaching of technical abilities, leading to reduced importance on the cognition of 
technology. These results can serve as a reference for further course design. 

The course also provided students with career exploration opportunities. Several graduates of High school A 
who attended the course have already entered good universities and technological universities with related 
departments (such as department of multimedia design, department of game creation, department of interior 
design...etc.) through college admission such as personal applications or recommendations by high schools, 
which corresponded to the originally established course objectives. 
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However, the course did not result in significant difference in the students’ interest in learning about technology. 
The researcher infer that the reason for this may have been that this course was an interdisciplinary elective 
course (among 10-12 courses) in High School A, and students who opted for the course had selected it after 
attending a course description seminar and fully understanding its content. Students who selected this course 
from among a variety of courses may have already had a high level of interest in virtual reality technology, so 
that although students’ interest in learning about technology was increased after the course, the increase was not 
significant. Therefore, the researcher suggests that future course teachers appropriately integrate teaching 
strategies that increase students’ learning motivation during the implementation of the course, such as adding 
current examples of application of virtual reality or 3D technology, and explaining more about future learning 
approaches and career development in related domains. This should assist in enhancing students’ interest in 
learning about technology. 

In external evaluation of this course, Huang et al. (2009) pointed out that compared to mainstream virtual reality 
technology, the resources and facilities of high schools are limited and high school students must be seen as 
beginning learners in dynamic digital technology. Therefore, the arrangement of subject matter, learning 
process, and technological content must naturally correspond to practicality and feasibility. Therefore, based on 
the course development and evaluation in the first year, the course in the second year was modified to reduce 
focus on abstract theory in learning. The timetable of the course was modified to delay teaching such theory 
until after students had a certain amount of practical operational experience with virtual reality, which would 
naturally make subsequent learning more meaningful. Also, to enhance students’ interest in learning about 
virtual reality and provide them with a preliminary understanding of related university departments, in addition 
to visit to nearby universities (or technological universities) with related departments, The researcher suggests 
that visits to multimedia companies can be added to the course in the future, in order to provide students with a 
more complete vision. 

Performance assessment, whether it involves Q&A sessions with students or teacher assessment, is usually very 
time-consuming. With regard to student work in design-related fields, students must not only use the skills and 
principles learned but also incorporate their individual subjective consciousness, to order to comprehensively 
express the idea of their work. When teachers assess such work, apart from performing evaluation from the 
angles of academic principles and facts, they also use their individual and extremely subjective viewpoint and 
ideas in assessment. Therefore, performance assessment is often discriminated against. 

Although performance assessment is costly, time-consuming, and difficult to conduct, when viewed from the 
angle of its principles and methods, it still offers many advantages; for example: emphasizing students’ practical 
performance, shifting from previous methods of assessment that overly emphasized written examination, 
guiding students to take initiative in learning, and emphasizing teacher-student interaction in the instructional 
process. Therefore, education authorities should still encourage teachers to use performance assessment. At the 
same time, if an objective and valid assessment scale was provided, teachers could perform evaluation according 
to existing assessment indicators and scores. Before creating their work, students would also be able to 
understand which key points should be a focus. After assessment, students could use their score on the scale as a 
form of feedback to understand their strengths and weaknesses. 

This study conducted correlation analysis of the six dimensions (communication of theme, creativity, color, 
technique, structure, and comprehensiveness) in the assessment scale. Results showed that the dimensions were 
moderately correlated, and the six dimensions with the total score of the scale are significantly correlated, 
indicating a high internal consistency. However, when designing the proportional scores for the six dimensions 
of the scale, this study was limited by research time and the number of research targets; therefore, the weights 
were determined according to the personal experience of each researchers. Thus, whether the proportional 
specific weight of each dimension is appropriate must be determined through scientific and objective analysis. 
In the future, if further research could be conducted on the proportional specific weight of the scale dimensions, 
this would make the indicators in the performance assessment scale more complete. 
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Appendix. Performance assessment scale for the virtual reality course 


Name: 

Final score: 

Item 


Assessment indicators 


Score 



- 

+ 

+ + 

Completed 

a. Can hand in work assignments on time according to teacher instructions 



10 

work on 

b. Can complete work before deadline after limited reminders from the teacher 

5 

7 

9 

time 

(10%) 

c. Work is not completed without constant urging and supervision from the 
teacher 

0 

1 

3 



a. The work and the theme closely interlock 

20 

23 

25 


Communication of 

b. Does not have a good grasp of the theme; work does 
not fully express the theme 

10 

13 

17 


theme (25%) 

c. Considerable distance exists between the finished work 
and its theme; ability to communicate theme requires 
improvement 

1 

4 

7 



a. Work is creative and unique 

11 

13 

15 



b. Work has some uniqueness but is not outstanding 

7 

8 

9 


Creativity (15%) 

c. Additional guidance, examples, or demonstration is 
needed from the teacher before the student can develop 
his/her own ideas and creativity 

1 

3 

5 



a. Use of colors is appropriate; does not often use concept 
colors, giving the image a rich feeling 

8 

9 

10 


Color (10%) 

b. Skill in color use could be improved; richer color 
functions could be used to enhance the work 

5 

6 

7 



c. More guidance from the teacher in techniques of color 
use is needed to enrich this work 

1 

2 

3 

Work 

assignment 


a. Student is proficient in use of software components and 
understands the strengths and limitations of software 

8 

9 

10 

(80%) 

Techniques (10%) 

b. Can use basic components, but creative skills need 
improvement 

5 

6 

7 



c. Needs more guidance form the teacher to understand 
the strengths and limitations of the software and 
become proficient in these skills 

1 

2 

3 



a. Content is rich and comprehensive, corresponds to 
psychological development, presents aesthetic balance 

8 

9 

10 


Structure (10%) 

b. Has spatial feeling and expresses individual thought, 
but requires improvement in screen configuration 
relationships 

5 

6 

7 



c. Lacks in structure, requires more guidance form teacher 

1 

2 

3 


Comprehensiveness 

(10%) 

a. Understands creative concepts, construction process, 
and related principles; can make full use of software 
components 

8 

9 

10 


b. Can complete a project but needs improvement or 
remedial work in some areas 

5 

6 

7 



c. Work has some incomplete and unsatisfactory parts; 
insufficient use of software components 

1 

2 

3 

Attitude 

toward 

learning 

(10%) 


a. Willing to make class presentations, attentive in 
creative work, helpful and willing to instruct others 

8 

9 

10 

Creative process 
(10%) 

b. Can follow the teacher’s instructions, works hard, does 
not interact much with other students 

5 

6 

7 

c. Cannot work independently, requires constant 
supervision and goading from the teacher, rarely 
interacts with classmates 

1 

2 

3 

Comments 

from 

teacher 
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